System and method to indicate code block execution

ABSTRACT

In one implementation, computer executable instructions are provided to identify a group of instructions corresponding to a code block for an executable module. The instructions increment a counter associated with the code block to indicate the code block has been tested.

BACKGROUND

Testing of code generally has followed three traditional procedures. Oneprocedure involves unit testing where a number of languages includingJava, Python, and so forth encourage the designer to provide testprocedures that exercise a particular portion of code. Thus, it is up tothe designer to ensure that the unit tests completely cover allexecution paths. Another testing procedure involves code coverage wheretools are applied to determine code paths that have been traversedduring testing. Still yet another form of testing includes ad hoctesting, where the programmer performs their own informal testing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a code block verification system.

FIG. 2 illustrates an example of a code block verification system andexample test sequence.

FIG. 3 illustrates an example of a code block verification system wherecounters are stored in an external file.

FIG. 4 illustrates an example of a training phase and an intermediatephase for code block verification.

FIG. 5 illustrates an example of a post-testing phase for code blockverification.

FIG. 6 illustrates an example method for code block verification.

DETAILED DESCRIPTION

FIG. 1 illustrates an example of a code block verification system 100.The system 100 includes an executable module 110 that includes computerreadable instructions 120. The executable module 110 includes aplurality of code blocks shown as code block 1 at 130, code block 2 at140, through code block N at 150, where N is a positive integer. Thecode blocks 130-150 can represent various branches of computer readableinstructions 120 that operate in the executable module 110, where suchbranches are illustrated in accordance with a code path 160. Forexample, the code path 160 can represent different branches or decisionsthat are encountered during the execution of a program. Thus, if codeblock 1 at 130 were identified as branch A along the code path 160, codeblock 2 at 140 might be identified as branch B along the code path 160.In one example, if an IF statement were encountered e.g., IF conditionEQUALS TRUE, execute branch A beginning at code block 1 at 130 alongcode path 160, otherwise execute branch B beginning at code block 2 at140. Thus, when conditional or other statements are encountered withinthe executable module 110, respective code blocks 130-150 are identified(e.g., the beginning address of code block determined) for respectiveexecution branches along the code path 160. The code path 160 caninclude any number of branches, nested loops or other code constructs.

Various techniques can be employed to identify the code blocks 130-150for the executable module, such as a beginning address (or label) ofinterrupt routines, jump locations to other addresses in the executionmodule 110, conditional branches, or substantially any condition thatcauses the execution module to vector to a new code execution point orgroup of instructions along the code path 160. The code path 160 couldbe a serial path of branches with each branch being a group ofinstructions identified by the respective code blocks 130-150.Alternatively or additionally, the code path 160 can be structured likea hierarchical tree where some branches may follow one another and somebranches may be parallel and entered via various jumps, loops,interrupts, or other branch instructions

As used herein, in some cases, the term “code block” may refer generallyto a sequence of one or more non-branching machine/assemblerinstructions, with single entry and single exit points. A simple codeblock can consist of a single instruction, for example. Branch and jumpinstructions (e.g., for tests and subroutine calls) may not beconsidered to be included in code blocks as they instead may beconsidered to lie in between code blocks. Several branches can have thefirst instruction of a code block as their target—e.g., a design rulewhere a code block may be entered at the start and exit at the endthereof. Entries to a code block typically occur at the firstinstruction of the block. As desired, code blocks can be split up toensure this condition—e.g., that no branch target occurs after the firstinstruction for any non-trivial code block. Executable code can be“tiled” with combinations of branch and jump instructions and with codeblocks, for example.

When the code blocks 130-150 have been identified within the executionmodule 110, instrument counters 170 are utilized to associate variouscounters with code blocks, as demonstrated as counter 1 at 180, counter2 at 182, and counter C at 186, where C is a positive integer denotingthe number of counters. The instrument counters 170 represent code orother methods that associate the counters 180-186 with the respectivecode blocks 130-150. Such methods for example can include developerassociation of counters to code blocks, compiler association of countersto code blocks, or object code association of counters to code blocks,for example, as will be described below. The number of counters C can becorrelated to the number of code blocks N; however, not all code blocksneed to have a counter as some blocks may be identified as non-used codeand counters can also be implemented to verify if a set of predeterminedcounters increment in a specified order.

The counters 180-186 can represent whether or not an associated codeblock has been executed. Additionally, the value of a counter canindicate how many times a given code block has been executed. Forexample, if code block 1 at 130 were executed at least one time, thecounter 180 would show a positive value of 1 (greater than 1 if executedmore than one time). If the code block 2 at 140 were never executed,then its respective counter 2 at 182 would show a non-incremented value(e.g., zero). The counters 180-186 can be utilized to test and verifyexecution of the code blocks 130-150 of the executable module 110. Whilethe example of FIG. 1 demonstrates the counters 180, 182 and 186 asresiding within respective code blocks 130, 140 and 150, it will beappreciated that counters can be implemented outside of code blocks(see, e.g., FIG. 3).

As an example, by associating the counters 180-186 with identified codeblocks 130-150 and analyzing the counters after a desired amount oftesting, it can be determined whether or not the code blocks 130-150have been executed and therefore tested. Conversely, a counter for agiven code block can also be evaluated to determine if a code block hadnot executed during such testing. Thus, various efficiencies may beobserved by installing the counters 180-186 with the code blocks130-150. For example, if the counters 180-186 were not provided, it maybe difficult to ensure that code being executed during production runshas been tested. Also, it may be the case that programmers leave in codethat does not, or cannot, be executed. Such code may not therefore needto be tested and may be identified as such (e.g., identified asnon-production code and not instrumented with a counter). Further, itmay be difficult to demonstrate to others that code has been adequatelytested and thus, the use of counters 180-186 may provide a mechanism forsuch test verification.

By way of example, the counters 180-186 can be maintained within a datasegment (e.g., a modifiable data element)—and generally cannot belocated within read-only code segments unless the counter itself isread-only (not modifiable)—this can be a useful technique to incorporatethe code information in a post-training phase, for example. As anotherexample, the counters 180-186 can be maintained in an external filealongside the associated executable module 110 under test. The counters180-186 can also be maintained in a local database, in a remote databaseor the functionality of the counters can be distributed among local andremote databases, for example. Furthermore, although the counters180-186 are shown inside the respective code blocks 130-150, suchcounters could also be placed at a location remote to the code blocks.For example, executables in the code blocks 130-150 could make a remotecall to code that increments a counter outside of the code block ratherthan from within the confines of the code block executable itself.

As will be described herein, the executable module 110 can be providedas a computer readable medium having computer readable instructions 120.This includes a code block 130-150 that corresponds to a group ofinstructions for an executable module 110 and a counter 180-186installed with the code block 130-150 to indicate the code block hasbeen tested (e.g., via its execution during a testing phase). Theexecutable module 110 can be a hardware module or a computer readablemodule. For example, a computer readable module may include instructionsexecuted from a CD ROM. A hardware module may include logic such asprovided with a gate array or other programmable device. This couldinclude testing of code such as Virtual Hardware Description Language,for example. The counters 180-186 can be incremented initially during atraining phase or checked for a non-zero value during an intermediatetesting phase, where a warning can be generated if a zero value isdetected during such intermediate testing. The counters 180-186 can beutilized during a post-testing phase (e.g., after code has beenreleased). During such post-testing use, the executable module can beprogrammed to generate a warning and to stop code execution if a zerovalue is detected in a counter during runtime execution.

The counters 180-186 can be implemented according to various techniques.One example includes associating counters via source code modifications,such that the counters are installed with the code blocks. This couldinclude the designer providing counters and the associated verificationlogic during programming and identification of the respective branchesand code paths. Alternatively, source code modification could occurautomatically via a designer tool that automatically provides countersand associated logic and programmatically associates the counters withcode blocks while the designer was creating or editing the program. Inanother example of counter association, a compiler may identify codeblocks in a module and automatically insert counters and verificationlogic corresponding to the identified code blocks during compilation ofthe source code. For example, if an IF statement were encountered, thecompiler could automatically install counters 180-186 at each resultantbranch occurring from the IF conditional statement. In yet anotherexample of counter association, code blocks may be identified andcorresponding counters installed during object code execution. Forexample, a Low Level Virtual Machine (LLVM) could be employed to detectbranches and provide counters 180-186 and associated verification logicinto the object code. Counters also may be programmatically associatedwith respective code blocks during one or more of the code installationprocesses disclosed herein.

Counters 180-186 can be provided at various locations in the respectivecode blocks 130-150, example counter locations within a code block areshown. One possible location is at or near a code block entry point.Thus, a counter 180-186 could be incremented at or near the beginningaddress of a code block 130-150. In another example of counter location,an intermediate point within a respective code block 130-150 couldinclude a counter. One or more such counters 180-186 could be providedthroughout a code block as deemed appropriate. In another example,counters 180-186 could be provided at or near the exit point of codeblocks. Thus, before the final return statement for a routine isexecuted, a counter could be provided.

Beyond the single counters 180-186 described above, alternative counterexamples are possible. In one example, nested counters could beprovided. For example, a nested counter could be a counter thatincrements when two or more other counters are incremented. In anotherexample, a concurrency counter may be employed. Such counter may beemployed to test foreground and background operations of a program. Forexample, it may be desirable to know that during execution of a givencode block 130-150, which interrupt routines were also executed. Thus, aconcurrency counter could be incremented when a given code blockexecuted concurrently with some other routine.

For purposes of simplification of explanation, in the present example,different components of the system 100 are illustrated and described asperforming different functions. However, one of ordinary skill in theart will understand and appreciate that the functions of the describedcomponents can be performed by different components, and thefunctionality of several components can be combined and executed on asingle component. The components can be implemented, for example, ascomputer executable instructions (e.g., software, firmware), hardware(e.g., CPU, an application specific integrated circuit), or as acombination of both. In other examples, the components could bedistributed among remote devices across a network, for example.

FIG. 2 illustrates an example of a code block verification system 200and example test sequence. Similar to FIG. 1, the system 200 includes anexecutable module 210 that includes computer readable instructions 220.The executable module includes a plurality of code blocks shown as codeblock 1 at 230, code block 2 at 240, through code block N at 250, whereN is a positive integer. The code blocks 230-250 represent variousbranches of computer readable instructions 220 that operate in theexecutable module 210, where such branches are illustrated in accordancewith a code path 260. When the code blocks 230-250 have been identified(e.g., by a programmer or automatically by a computer) within theexecution module 210, instrument counters 262 are associated with thevarious code blocks and illustrated as counter 1 at 264, counter 2 at266, and counter C at 268, where C is a positive integer that isgenerally correlated to the number of code blocks, N. The counters264-268 represent whether or not a code block has been executed as wellas the number of times it has been executed. The system 200 includes aprocessing unit 270 to execute the computer readable instructions 220 inthe executable module 210. Such processing unit 270 can also be employedto execute functions corresponding to various test phases that aredisclosed herein.

When the counters 264-268 have been associated with the respective codeblocks 230-250, a testing phase 280 can be entered. The testing phase280 can include a training phase where the executable module 210 isinitially exercised and the counters 264-268 incremented if a givenbranch of the code path 260 has executed. A map generator 282 cangenerate a map of counters 284 that quantifies the values of thecounters 264-268 for each of the respective code blocks 230-250. The mapof counters 284 can include a testing profile or behavior map. Such mapcan be employed to determine normalized operations for a given block ofcode. Thus, if other code blocks are added in the future, the normalizedbehavior can be checked against the additions to verify previousfunctionality has remained substantially consistent.

A verification block 285 can analyze the map 284 to determine if anycode blocks have not been executed. If the verification block 285determines that any of the counters 264-268 have not been incremented(e.g., zero value), a re-test loop can be initiated (e.g., automaticallyby the verification block 285 of the testing phase 280 or manually by atest engineer) as shown at reference numeral 286. Re-tests can be run asmany times as desired or until a level of confidence has occurred (e.g.,re-test until all counters have non-zero value or 90% of counters havenon-zero value and so forth). After the training phase, intermediatephases of the testing phase 280 can be executed and reiterated at 286 asdesired.

After a level of confidence has developed with the intermediate testingphase 280, a post-testing phase can be entered as shown at 290. In thepost-testing phase the code is deemed to be release code 292 whichgenerally indicates the code has been shipped or provided to users ofthe respective code. As the code is executed by users of the releasecode 292, warnings can be generated at 294 if a non-incremented counteris detected during operation of the release code 292. Moreover, therelease code 292 can be designed to terminate execution if a counterthat was not incremented during testing is detected during execution ofthe release code 292. If such warning and/or halt of execution weregenerated at 294, the designer of the release code 292 can be notified(e.g., e-mail automatically generated, instant message opened, web pagelaunched) that a non-tested portion of release code 292 was encountered.The designer can then initiate further actions such as recall,re-testing, re-design, and so forth as deemed appropriate.

The system 200 can be provided as part of a testing platform coupled toa network, where the platform includes a memory for storing computerreadable instructions 220. The processing unit 270 accesses the memoryand executes the computer readable instructions 220. The computerreadable instructions 220 can include a code block 230-250 to provideinstructions associated with the executable module 210 and a counter264-268 associated with the code block to indicate the code block hasbeen executed during testing.

Counters can be handled differently depending on the phase or testsequence. In one example, hook calls may be employed to increment,maintain, or otherwise verify operation of a given counter andassociated code block. For example, a hook call can correspond to asubroutine call (typically with at least one argument to identify ituniquely). These can be provided as entry points into a given libraryand may be used to provide particular specialization code. Hooksubroutines, for example, can be implemented via some type of high-levelscript or via a supplied low-level code library. In the example system200, hook calls could be placed at the start of each block, such as by asource-code compiler and/or binary linker optimizing tool. Thisgenerally does not change the block structure itself but adds an extrabranching or subroutine call at the start of each code block. As anexample, such hook calls can be implemented via supplied low-levellibraries (e.g., one hook procedure library designated for use duringthe training phase 280 and another for use during the post-testing phase290).

Operation of the hook call may depend upon the mode of operation (e.g.either training or normal). During the training phase 280, the call tothe hook procedure could increment the appropriate block count, forexample. During normal operation, the hook procedure library could bereplaced with a library containing code to keep track of the counters.In each case, the call to the hook procedure could use an argument todetermine uniquely to which counter the call refers. The appropriateaction to take could be determined by supplying a different code libraryto implement the hook calls—depending upon the operating mode (e.g.,either training or normal). In normal operation, the hook calls checkthat the associated counter is non-zero and then allowing execution tocontinue onwards or otherwise take exceptional action. A potentialadvantage of the hook call approach is that a recompile of the originalsource may not be necessary to switch modes—thus, only the code libraryimplementing the hook subroutines may need to be changed.

FIG. 3 illustrates an example of a code block verification system 300where counters reside external to the code blocks. Similar to FIG. 1above, the system 300 includes an executable module 310 that includescomputer readable instructions 320. The executable module 310 includes aplurality of code blocks shown as code block 1 at 330, code block 2 at340, through code block N at 350, where N is a positive integer. Thecode blocks 330-350 can represent various branches of computer readableinstructions 320 that operate in the executable module 310, where suchbranches are illustrated in accordance with a code path 360. Forexample, the code path 360 can represent different branches or decisionsthat are encountered during the execution of a program as describedabove.

Counters, shown as counter 1 at 380, counter 2 at 382, and counter S at386, are associated with respective code blocks 330, 340 and 350. Whenthe code blocks 330-350 have been identified within the execution module310, the counters 380, 382, and 386 can be installed in a file 390 thatis separate from the code blocks 330, 340 and 350. The number ofcounters S can be correlated to the number of code blocks N; however,not all code blocks may have a counter. For instance, some code blocksmay be identified as non-used code. The association of counters 380, 382and 386 with respective code blocks 330, 340 and 350 can be userconfigurable. Additionally, counters can be implemented to verify if aset of predetermined counters increment in a specified order. Thecounters 380-386 can provide values that represent whether or not arespective code block has been executed. Additionally, the value of acounter can indicate how many times a given code block has beenexecuted. For example, if code block 1 at 330 were executed at least onetime, the counter 380 would show a positive value of 1 (greater than 1if executed more than one time). If the code block 2 at 340 were neverexecuted, then its respective counter 2 at 382 would show anon-incremented value (e.g., zero).

FIG. 4 illustrates an example of a training phase 410 and anintermediate phase 414 for code block verification. As shown, before thetraining phase 410 begins, source code blocks are instrumented withcounters at 420 and subsequently compiled. Alternatively, the counterscan be installed during compilation and/or during object code execution.After an application has been compiled at 420, it enters the trainingphase 410. During this phase 410, in response to executing a code block,a counter unique to the code block is incremented as shown at 430. Atthe end of training phase 410, the code enters production in theintermediate phase 414 in which the counter of each code block that hadexecuted has incremented to a non-zero value commensurate with thenumber of times it had executed. However, the training phase may not becompletely trusted. The code blocks are then exercised during theintermediate testing phase, as shown at 440. In response to executing acode block, the unique counter associated with the block is evaluated(e.g., by the verification block 285 of FIG. 1) to determine if it isnon-zero as illustrated at 450. If the counter is zero, then the codewas not traversed during its training phase 410. In this case, a warningcan be generated. For example, the warning issued at 460 can identifythe source code and line number for the non-executed code block. Furtherre-testing can commence at 470 where the code re-enters the trainingphase 410 or reiterates within the intermediate phase 414. Suchreiterations or testing can be continued until a desired level ofquality is achieved (e.g., one hundred thousand runs have been executedand no non-incremented counters were detected).

FIG. 5 illustrates an example of a post-testing phase 500 for code blockverification. As shown, at some subsequent stage of code development andtesting, the code is deemed as released code 510. Thus, the behavior ofthe intermediate phase code described above is changed where countersare checked at 520 and entering an untested block can result intermination of the process at 530. For example, a verification block canbe installed with the released code to determine if a counter for agiven code block increments for the first time during execution of thereleased code. A warning can be generated at 530, a warning withtermination of execution, or merely termination of execution can occurat 530 if a non-incremented counter (e.g., having a zero value) isdetected during execution of the released code 510. If thenon-incremented counter is encountered during the post-testing phase at530, automated messages can be generated to contact the manufacturer ordesigner, which can take appropriate action such as recalling theproduct and retesting before generating a subsequent version of thereleased code 510, issuing a software patch or the like.

In view of the foregoing structural and functional features describedabove, an example method will be better appreciated with reference toFIG. 6. While, for purposes of simplicity of explanation, the method isshown and described as executing serially, it is to be understood andappreciated that the method is not limited by the illustrated order, asparts of the method could occur in different orders and/or concurrentlyfrom that shown and described herein. Such method can be executed by acomputer and associated equipment, for example.

FIG. 6 illustrates an example method 600 for code block verification.The method 600 includes identifying a code block corresponding to agroup of instructions for an executable module at 610. As describedpreviously, this can involve analyzing for branches, loops, interrupts,conditional statements, and so forth to determine where a new executionpath begins. The identifying of code blocks can be implemented manually(e.g., by a programmer), automatically (e.g., by a computer) or acombination of manual and automatic mechanisms can be used. At 620, themethod 600 includes installing a counter with the code block. Forexample, such installation can be performed by source code modification,complier modification of the source code, and/or object codemodification as disclosed herein. The counter can be programmed toincrement when the respective code block executes. At 630, the method600 includes exercising the executable module, such as operating theexecutable module during a training or intermediate testing phase. At640, the method 600 includes determining if the code block has beentested by analyzing the counter. The counter may have been incrementedin a training phase when its associated code block is executed during atesting phase, such as disclosed herein. As described above, determiningif a code block has been tested generally includes analyzing the counterfor zero and non-zero values, where a zero value indicates a code blockhas not been tested and a non-zero value indicates a code block has beentested.

What have been described above are examples. One of ordinary skill inthe art will recognize that many further combinations and permutationsare possible. Accordingly, the disclosure is intended to embrace allsuch alterations, modifications, and variations that fall within thescope of this application, including the appended claims. As usedherein, the term “includes” means includes but not limited to, the term“including” means including but not limited to. The term “based on”means based at least in part on. Additionally, where the disclosure orclaims recite “a,” “an,” “a first,” or “another” element, or theequivalent thereof, it should be interpreted to include one or more thanone such element, neither requiring nor excluding two or more suchelements.

1. A computer readable medium comprising computer executableinstructions that when executed cause a processor to: identify a groupof instructions corresponding to a code block of an executable module;and increment a counter associated with the code block to indicate thecode block has been tested.
 2. The computer readable medium of claim 1,further comprising computer executable instructions to increment thecounter during a training phase in response to the code block beingexecuted.
 3. The computer readable medium of claim 2, further comprisingcomputer executable instructions to verify the counter for a non-zerovalue during an intermediate testing phase and to generate a warning ifa zero value is detected.
 4. The computer readable medium of claim 3,further comprising computer executable instructions to check the counterduring post-testing run-time operation and to stop code execution if azero value is detected.
 5. The computer readable medium of claim 1,further comprising computer executable instructions to implement anested counter or a concurrency counter to monitor execution of a codepath combination comprising multiple code blocks.
 6. The computerreadable medium of claim 1, further comprising computer executableinstructions to implement a map generator to generate a map from aplurality of counters, corresponding to respective code blocks in theexecutable module, to verify performance of the executable module or togenerate a test behavior profile.
 7. The computer readable medium ofclaim 1, wherein the counter is located near an entry point to the codeblock.
 8. A method, comprising: identifying a code block correspondingto a group of instructions for an executable module; installing, by aprocessor, a counter for execution with the code block that has beenidentified for the executable module; exercising the executable module;and determining, by the processor, if the code block has been tested byanalyzing the counter.
 9. The method of claim 8, further comprisinggenerating a map to determine a testing behavior for the code block. 10.The method of claim 8, further comprising terminating code execution ifa non-incremented counter is detected after the testing.
 11. The methodof claim 8, further comprising generating a warning if a non-incrementedcounter is detected after testing.
 12. The method of 8, furthercomprising associating the counter with the code block via a source codemodification.
 13. The method of 8, further comprising associating thecounter with the code block via a modification by a compiler.
 14. Themethod of 8, further comprising associating the counter with the codeblock via an object code modification by a virtual machine.
 15. A codetesting system comprising: a memory storing computer executableinstructions, the memory comprising code blocks to correspond torespective groups of instructions of an executable module; and aprocessing unit to access the memory and execute the computer executableinstructions, the computer executable instructions, when executedcausing the processing unit to: increment a counter to provide anindication of testing of a code block during a training phase; generatea warning if the counter indicates that the code block has not beenexecuted during the training phase; terminate code execution if thecounter indicates that the code block has not been tested; and generatea map from a plurality of counters for respective code blocks to provideinformation about testing of the code blocks.